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CHAPTER  7 

TOPOLOGICAL  DESIGN  _OF  GATEWAYS  FOR  PACKET 
SWITCHED  INTER-NETWORK  COMMUNICATIONS 


7.1.  INTRODUCTION  AND  SUMMARY 


In  this  Chapter  we  consider  the  topological  design  issues 
associated  with  packet-switched  network  intercommunication.  In 
particular,  we  present  an  effective  (heuristic)  solution  procedure 
for  a simplified  internet  topological  model,  and  illustrate  the 
design  tradeoffs  by  applying  the  procedure  to  the  interconnection 
a multi-network  example,  and  the  interconnection  of  ARPANET  and 
a ^L^nplified  model  of  AUTGDIN  II  packet-switched  backbone  network. 

^Internetwork  communication  has  become  a,  increasingly  active 
area  of  research  in  the  last  few  years.  However,  up  to  now,  most 
research  efforts  have  been  concentrated  on  the  protocol  and  gateway 
design  issues  J^fcERF , 1974],  [CERF,  1975],  [ BBN , 1975].  ft  The  purpose 

of  this  study  is  to  investigate  the  topological  design  related 
issues.  v” 

In  Section  7.2,  a brief  survey  on  the  current  work  in  network 
intercommunication  is  given.  A general  approach  to  establish  inter- 
net communication  is  to  co;.nect  the  networks  through  "gateways",  or 
"gateway-hal\ es"  (GH) . The  gateway-halves  together  with  the  (actual 
or  pseudo)  links  connecting  them  form  a so-called  "gateway-half 
virtual  network".  A fundamental  criterion  in  internet  communica- 
tion is  that  the  internal  operations  of  the  individual  networks 
should  not  be  interfered  with.  Based  on  this  so-called  independence 
principle,  some  general  requirements  can  be  deduced.  For  example, 
to  allow  the  local  net  better  protection  against  the  internet  traffic, 
the  gateway-halves  should  connect  to  the  networks  at  the  Host  level; 
also,  an  internet  packet  should  appear  as  a locaj  packet  in  the 
local  nets.  Several  internet  end-to-end  protocol  issues  are  addres- 
sed. The  first  issue  is  whether  internet  packet  fragmentation 
should  be  done  at  the  gateway-halves.  Other  issues  related  to 
internet  flow  control  and  error  control  are:  Should  flow  control  be 
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performed  on  letters  or  on  fragments?  Should  error  control  be  made 
optional?  Should  retransmisssion  between  gateway-nalves  be  used? 

Each  of  the  issues  is  discussed  briefly.  We  then  list  the  basic 
gateway-ha] f functional  requirements:  message  processing  between 

two  networks,  dntergateway  routing,  access  control,  accounting, 
statistics  gathering,  internetwork  message  f ragmentat i c:« , congestion 
control  and  intergateway  retransmission. 

In  Section  7.3  we  present  a formulation  of  the  general  internet- 
work topological  design  problem,  and  indicate  of  the  inherent  dif- 
ficulties. We  then  examine  the  alternatives  in  interne'  routing  and 
network  design.  The  important  issues  are:  Should  a fixed  or  an 

adaptive  routing  policy  be  used?  Should  the  Hosts  participate  in 
route  selection?  Should  a single  level  or  a hierarchical  routing 
policy  be  used?  Can  the  local  requirements  be  routed  through  the 
internet  and  use  other  networks'  resources?  Can  the  loca^  net  topo- 
logy and  link  capacities  be  modified  in  the  internet  design  process? 
For  some  of  the  issues,  reasonable  decisions  can  be  made  based  on 
the  local  net  independence  principle  and  previous  single  network 
design  experience,  e.g.,  hierarchical  adaptive  routing  policies  are 
preferred  over  single  level  fixed  routing  policies.  For  other 
issues,  the  answers  are  not  so  clear,  and  for  some  the  non-technical 
considerations  may  be  the  dominant  factors  in  decision-making.  An 
internet  routing  model  is  presented  in  Section  7.4.  The  routing  of  an 
internet  packet  is  carried  out  in  three  stages. 


1.  Local  net  routing  from  the  source  switch 
(or  Host)  to  the  source  regional  gateway- 
half. 

2.  Gateway-half  virtual  net  routing  from  the 
source  regional  gateway-half  to  the  desti- 
nation regional  gateway-half. 

3.  Local  net  routing  from  the  destination  re- 
gional gateway-half  to  the  destination 
switch  (or  Host) . 

7.2 


J* 

y. 


r-il  ■ -Tr  r 


fN  c l wo/nrv  atv mi_  t o i o twnrunM 


0/1* 


With  this  routing  model,  the  original  Host-to-Host  traffic  require- 
ments reduce  to  the  following  three  types:  Local  net  switch  to 

switch  requirements,  local  net  switch  to  gateway-half  requirements 
induced  from  the  internet  requirements,  gateway-half  virtual  net  to 
gateway-half  requirements,  induced  from  the  internet  requirements. 

The  internet  and  local  end-to-end  delay  constraints  can  then  be 
reformulated  in  terms  of  these  three  types  of  requirements. 

Based  on  the  internet  routing  model,  a simplified  internetwork 
topological  design  problem  is  formulated  in  Section  7.5.  A design 
procedure,  using  extensions  of  existing  single  net  design  techniques, 
is  presented.  Briefly,  the  procedure  is  as  follows:  First,  based 

on  the  internet  traffic  requirements  and  the  local  net  characteris- 
tics, a set  of  gateway-halves  is  selected  from  each  net.  The 
routes  for  the  loca]  (switch  to  switch  and  switch  to  gateway-half) 
requirements  are  then  determined  in  each  local  net.  If  allowed,  the 
local  net  topologies  and  link  capacities  are  also  optimized.  An 
estimate  on  the  incremental  delay  per  unit  incremental  flow  can  then 
be  determined  between  each  pair  of  gateway-halves  residing  in  the 
same  local  net.  This  information  is  used  for  the  gateway-half 
virtual  net  routing  and  topological  design.  The  procedure  can  be 
applied  iteratively  to  obtain  design  improvements.  A procedure  for 
selecting  the  gateway-halves,  based  on  methodologies  similar  to 
backbone  switch  selection  procedure  [NAC,  1976a}  and  satellite 
switch  selection  procedure  [NAC,  1976b]  is  also  presented. 

The  design  procedures  have  been  implemented  on  a PDP-10  based 
on  modifications  and  extensions  of  existing  NAC  single  network, 
single  traffic  class  interactive  design  system.  Due  to  the  multi- 
network, multi-traffic  classes  nature  of  the  internetwork  design 
problem,  the  modifications  are  substantial.  These  programs  have 
been  applied  to  study  the  interconnection  problem  of  a mul ti-network 
model  and  the  interconnection  of  ARPANET  and  AUTODIN  II.  Below 
we  briefly  summarize  the  main  findings.  Detailed  design  results 
are  reported  in  Section  7.6. 
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On  the  multi-network  model,  studies  were  carried  out  based  on 
the  following  assumptions: 

1.  The  sum  of  the  local  requirement  and  the 
internet  requirement  is  constant; 

2.  The  local  traffic  requirement  is  dis- 
tributed uniformly  among  the  local  switch 
pairs ; 

3.  The  internet  traffic  requirement  is  dis- 
tributed uniformly  among  the  internet 
switch  pairs. 


"'^e  following  observations  emerge: 


1.  The  higher  the  proportion  of  internet  traffic 
requirement,  the  higher  the  total  system  cost. 
(Note  that  the  total  traffic  is  constant) . 

2.  Suppose  each  local  net  is  optimally  designed 
for  the  local  traffic,  and  is  not  allowed  to 
change.  Then  in  order  to  accommodate  the 
internet  traffic  (even  under  the  assumption 
that  total  amount  of  local  and  internet  re- 
quirement is  const",nt)  , gateway-halves  are 
required  at  at  least  half  of  the  switch 
locations. 


3.  Suppose  only  a moderate  number  of  gateway-halves 
are  used.  By  allowing  modifications  on  the 
local  net  link  capacities  (but  not  topologies), 
good  internet  design  can  be  obtained. 
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4.  The  total  cost  for  an  interconnected  system  is 
of  the  same  order  as  the  total  cost  for  a fully 
integrated  system,  assuming  that  no  extra  pro- 
tocol overhead,  no  software  modification  cost 
is  required  for  the  later  system. 

On  the  ARPANET-AUTODIN  II  model,  studies  were  carried  out  assuming 
fixed  local  net  topologies  (link  capacities  may  be  modified) , and 
fixed  local  net  traffic  requirements.  The  peculiarity  of  this 
system  is  that  the  ARPANET  traffic  level  and  utilization  are  very 
low  but  the  number  of  switches  is  large.  Studies  were  carried  out 
to  determine  the  maximum  possible  internet  throughput  levels  for 
various  number  of  gateway-halves,  under  the  assumption  that  neither 
ARPANET  nor  AUTODIN  II  is  allowed  to  change.  Figure  1 shows  the 
maximum  internet  throughput  as  a function  of  the  number  of  gateway- 
halves  used  in  the  ARPANET.  Moreover,  it  was  observed  that  by  al- 
leging modification  in  local  net  link  capacities,  only  minor  cost 
savings  (5%)  can  be  achieved  at  the  base  unit  costs  ($5/mile/mo.  for 
50  Kbps  trunk  lines,  and  $2K/mo.  for  the  gateway-half  modules)  . By 
pax ametrizing  the  unit  component  costs,  it  was  observed  that  the 
cost  differences  between  designs  with  local  nets  fixed  and  designs 
with  local  nets  changeable  are  quite  sensitive  to  changes  in  unit 
communications  cost,  but  less  sensitive  to  changes  in  unit  gateway- 
half  module  cost.  When  the  cost  of  gateway-halves  increases,  the 
cost  differences  between  the  two  types  of  designs  becomes  more 
apparent.  Figure  2 shows  the  cost  differences  between  the  fixed 
local  net  designs  and  changeable  local  net  designs,  for  different 
internet  throughput  levels  and  different  unit  component  costs. 

The  cost  difference  between  the  two  options  (fixed  or  modifiable 
local  nets)  ranges  from  1%  to  12%.  For  low  gateway-half  cost, 
the  cost  difference  is  more  sensitive  to  communication  cost  than 
to  internet  throughput  level.  When  the  gateway-half  cost  is  in- 
creased, then  the  cost  difference  becomes  sensitive  to  the  inter- 
net throughput  level. 
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NUMBER  OF 
GATEWAY-HALVES 
IN  ARPANET 


FIGURE  1:  INTERNET  THROUGHPUT  VS.  NUMBER  OF  GATEWAY-HALVES  USED  IN  ARPANET: 

ARPANET  - AUTODIN  II  DATA  BASE;  FIXED  LOCAL  NETS 
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TOTAL  SYSTEM  COST 


100  200  300  400  500  600 

SYSTEM  COST  WITH  LOCAL  NETS  FIXED 

SYSTEM  COST  WITH  LOCAL  NETS  CHANGABLE 

FIGURE  2 : TOTAL  SYSTEM  COST.  COMPARISON  BETWEEN  DESIGNS 

WITH  LOCAL  NET  FIXED  AND  DESIGNS  WITH  LOCAL 
NETS  MODIFIABLE:  ARPANET  - AUTODIN  II  DATA  BASE 
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7.2.  BASIC  FRAMEWORK  FOR  NETWORK  INTERCONNECTION 


In  this  section  we  present  the  basic  framework  and  the  rele- 
vant protocol  and  gateway  issues  for  internetwork  communication. 
General  surveys  on  network  interconnection  can  be  found  in  [CERF, 

1974]  [SUNSHINE,  1977]  [ROBERTS,  1976]  [BBN , 1975]. 

7.2.1  Basic  Entities  in  Internetworking 

First,  we  briefly  examine  the  elements  of  a packet-switched 
network  (Figure  3) . A packet-switched  network  is  composed  of  a set 
of  computer  resources  (Hosts),  a set  of  one  or  more  packet  switches 
(network  nodes),  and  a set  of  communication  channels  (network  links) 
that  interconnect  the  packet  switches.  A Host  may  have  several 
active  processes  which  communicate  with  processes  in  the  Host  and 
processes  on  other  Hosts.  It  is  assumed  that  each  Host  will  con- 
tain a transport  station  responsible  for  multiplexing  the  interface 
with  the  communication  subnet,  and  "adding  value"  to  the  packet 
switching  service.  For  identification  purposes,  we  suppose  that 
each  process  communicates  through  one  or  several  ports  in  the  trans- 
port station.  Each  port  is  assigned  a unique  name  (or  address)  in 
the  multi-network  context  (i.e.,  even  though  the  addresses  for  the 
ports  may  change  dy  amically,  at  any  moment  no  two  ports  can  have 
the  same  address) . Thus  a transport  station  can  be  considered  as  a 
collection  of  ports.  The  communication  between  a source  port  and  a 
destination  port  is  effected  by  an  association  between  the  two 
ports. 

The  communication  subnet  of  a packet-switched  network  usually 
exhibits  the  following  common  operational  characteristics:  [McKENZIE, 

1974] 

• Although  the  average  delay  between  accepting  and 
delivering  a packet  is  relatively  small,  the 
variance  may  be  large. 
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FIGURE  3:  A PACKET-SWITCHED  NETWORK 
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Packets  may  be  delivered  in  an  order  different 
from  that  in  which  they  were  accepted. 

Duplicate  copies  of  packets  may  be  delivered. 

Some  packets  may  not  be  delivered,  (with  a 
non-neglibible  probability) . 

Bit  error  probability  in  delivered  packets 
is  small. 


However,  individual  packet-switched  networks  may  be  different  in 
implementation  as  follows: 


Each  network  may  have  distinct  w ays  of  address- 
ing the  receiver. 

Each  network  has  its  own  routing,  fault  detec- 
tion, flow  control  techniques. 

Each  network  has  its  own  format  of  packets  and 
its  own  Host/node  and  Host/Host  protocols.  The 
maximum  packet  size  may  be  different  in  different 
networks. 


Each  network  has  its  own  time  delay  character- 
istics in  accepting,  delivering  and  transport- 
ing the  data. 


The  interconnection  of  a pair  of  (or  several)  packet-switched 
networks  can  be  done  in  one  of  two  ways  (see  Figure  4): 


7.10 


NETWORK  ANALYSIS  CORPORATION 


\ PS 

\ n> 


(A)  GATEWAY  CONNECTION  MODEL 


\ 

PS  GH  \ 


\ GH  PS 
\ 


GH  PS 
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FIGURE  4:  INTERNET  CONNECTION  ALTERNATIVES 
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1.  Via  gateway  units  which  act  as  hosts  to  two 
(or  several)  networks, 

2.  Via  connection  through  one  or  several  pairs 
of  gateway-halves  (GH's)  each  residing  as  a 
host  in  one  network. 

The  merits  and  weaknesses  of  the  Gateway  and  Gateway  Halves  approaches 
as  well  as  the  functions  to  be  included  in  these  devices  have  been 
explored  in  (BBN , 1975],  [CERF,  1974],  [DAY,  1975],  [INWG,  1975], 
[McKENZIE,  1974],  [POUTIN,  1974]  and  others,  and  is  not  elaborated 
upon.  In  this  chapter,  the  discussion  will  be  carried  out  under 
the  GH  model,  although  most  of  the  results  are  equally  valid  under 
the  gateway  model. 

The  gateway-halves,  together  with  the  interconnecting  links, 
form  a "gateway-half  virtual  network"  (see  Figure  5).  The  GH's 
residing  in  different  networks  are  connected  by  intergateway  links 
in  order  to  effect  internetwork  communication.  Some  of  the  GH's 
residing  in  the  same  network  may  also  be  connected  by  intergateway 
links  in  order  to  effect  fast  transmission.  Moreover,  the  GH's 
residing  in  the  same  network  are  connected  by  "pseudo-links"  which 
correspond  to  local  net  paths  between  the  GH's.  Thus,  the  GH's 
residing  in  the  same  network  together  with  the  interconnecting 
pseudo-links  form  a complete  graph. 


7.2.2 


Basic  Operational  Requirements  for  Internetworking 


Because  packet-switching  technology  is  still  in  its  early 
development  stage,  it  may  be  premature  to  force  a common  standard 
on  '.11  the  networks.  Hence,  the  fundamental  principal  in  inter- 
net ork  communication  is  that  it  should  not  interfere  with  the  in- 
ternal operation  of  the  individual  networks  (even  though  some  net- 
works may  choose  to  modify  their  operations  to  making  internetwork- 
ing easier) . Thus  the  individual  network  differences  mentioned 
earlier  should  be  taken  into  account. 

7.12 


NETWORK  ANALYSIS  CORPORATION 


intergateway  lin 
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FIGURE  5:  GATEWAY-HALF  VIRTUAL  NETWORK 
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Based  on  the  local  net  independence  principle,  some  general 
requirements  for  internetworking  can  be  stated:  [CERF,  1974]  [SUN- 

SHINE, 1977]  [MCKENZIE,  1974] 

1.  The  GH's  should  connect  to  the  networks  at  the 
Host  level  (or  even  user  level).  In  this  way, 
each  network  can  protect  itself  against  the 
activities  of  the  GH's  to  the  same  extent  as 
it  may  protect  itself  against  the  activities 
of  any  other  Hosts. 

2.  Because  in  general  it  is  difficult  to  convert 
between  the  Host-to-Host  protocols  of  two  dif- 
ferent networks,  the  GH's  should  communicate 
with  nodes  of  the  network  in  the  lowest  form 
of  Host/node  protocol  supported  by  the  net- 
work. 

3.  A common  (international)  protocol  is  required 
for  Hosts  desiring  internetwork  communication. 

Note  that  a Host  may  thus  have  to  implement 
two  sets  of  Host-to-Host  protocols,  one  for 
its  own  network,  and  one  for  internetworking. 

4.  A uniform  addressing  scheme  is  required  which 
can  be  known  to  all  Hosts  and  processes  par- 
ticipating in  internetworking. 

5.  A common  (international)  packet  format  is  re- 
quired. However,  within  each  individual  net- 
work, the  international  traffic  must  appear 
the  same  as  the  local  traffic.  The  common 
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technique  used  is  to  embed  the  internetwork 
segment  inside  the  local  packet.  (See  Figure  G) . 
With  suitable  standardization  in  local  packet 
format,  the  local  header  and  the  internet  header 

l 

can  share  some  common  segments,  thus  reducing 
the  overhead  [CERF,  1975]. 

6.  Because  of  the  differc.  in  maximum  packet  size 
allowed  in  different  networks,  either  all  inter- 
network messages  have  to  transmit  in  units  of 
the  smallest  maximum  size  (which  at  present  is 
roughly  2000  bits)  or  data  crossing  network 
boundaries  has  to  be  fragmented  (by  the  GH's) 
into  smaller  pieces.  Possible  fragmentation 
schemes,  are  discussed  in  [CERF,  1974],  [MCKENZIE, 
1974]  . 

7.2.3  End-to-End  Protocol  Issues 


Several  important  end-to-end  protocol  issues  which  have  been 
researched  in  [INWG,  1975],  [CERF,  1975],  [BBN , 1975],  are  briefly 
presented. 


A.  Fragmentation  Between  Gateways 


As  mentioned  in  Section  7.2.2,  different  networks  may  have 
different  maximum  packet  sizes.  Hence,  either  all  internetwork 
traffic  must  be  transmitted  in  standard  size  packets  allowed  by  all 
networks,  or  some  fragmentation  mechanisms  are  required  at  the 
gateway-halves,  so  that  messages  from  one  network  can  be  fragmented 
into  smaller  pieces  for  transmission  through  next  network. 
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FIGURE  6:  INTERNETWORK  PACKET  FOPMAT 
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Some  of  the  arguments  for  fragmentation  at  the  gateways  aie: 

• New  high  speed  technology  and  cheap  memories 
make  it  attractive  to  use  packets  longer  than 
2000  bits  in  some  networks,  but  networks  built 
with  older  technology  may  still  be  limited  to 
the  smaller  length. 

• It  is  inefficient  to  force  all  networks  to  use 
only  the  minimum  size  that  all  networks  can 
guarantee  to  carry.  Moreover,  It  could  be  harm- 
ful to  bind  future  network  designs  by  the  present 
technology  standards. 

• Encryption  procedures,  to  function  at  all,  may 
require  fragmentation  at  the  gateway. 

Some  of  the  arguments  against  fragmentation  at  the  gateways 

are : 

• Public  networks  will  use  a standard  size  for 
datagrams,  and  private  networks  will  tend  to 
adopt  the  same  standards,  at  least  as  an  option. 

• Practically  all  current  networks  carry  fragments 
at  least  2000  bits  in  length.  (However,  it  should 
be  noted  that  in  ARPANET,  the  maximum  packet  length 
is  only  1000  bits) . 

• There  will  always  be  a large  proportion  of 
letters  shorter  than  2000  bits  for  transaction 
oriented  traffic. 
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• .ving  longer  messages  or  packets  to  improve 

network  efficiency  should  not  be  the  respon- 
sibility of  the  user.  Rather,  it  should  be  a 
multiplexing  scheme  introduced  within  or  at 
the  packet  switched  network  interface  level, 
wherever  useful  (e.g.,  as  in  TYMNET  and  ARPANET). 

It  is  recognized  in  [INWG,  1975]  that  if  gateway  fragmentation 
is  used,  a mechanism  should  be  provided  whereby  the  receiving  trans- 
port station  can  specify  the  maximum  fragment  size  it  is  willing  to 
receive.  (It  may  happen  that  the  maximum  packet  size  a transport 
station  can  handle  is  smaller  than  the  maximum  allowable  packet  size 
of  the  network  it  resides) . 

B.  Flow  Control,  Error  Control 


There  are  several  related  issues: 


1.  Should  flow  control  be  performed  on  lecters  or 
on  fragments? 

2.  Should  error  control  be  made  optional? 

3.  Should  retransmission  between  gateway-halves 
be  used  (possibly  as  an  option)? 


The  issue  of  whether  flow  control  should  be  done  on  letters  or 
on  fragments  is  related  to  both  the  Host  buffer  management  and  the 
communication  link  utilization.  If  the  probability  of  retransmis- 
sion is  small,  then  it  can  be  shown  that  the  increase  in  ] ink  utili- 
zation using  the  fragment  retransmission  scheme  over  the  letter  re- 
transmission scheme  is  rather  insignificant.  However,  a fragment- 
oriented  acknowledgment  retransmission  scheme  should  result  in  more 
flexib3e  Host  buffer  allocation,  and  hence  better  utilization. 
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There  is  also  a suggestion  of  using  a "fragment  count"  on  letters 
(thus  still  using  a letter-based  control  mechanism) , which  should 
also  make  the  buffer  management  more  efficient. 

Next,  consider  the  issue  of  whether  error  control  should  be 
made  optional.  Some  users  may  already  have  their  own  error  control 
scheme,  and  thus  all  they  need  is  a simple  letter  switching  service 
and  no  network  level  error  control  is  required.  However,  it  is  ar- 
gued that  these  types  of  users  are  not  common,  and  they  probably 
need  some  flow  control.  Hence,  for  the  preliminary  protocol 
designs,  it  may  be  simpler  to  consider  error  control  and  flow 
control  as  a standard  basic  service. 

Finally,  consider  the  issue  of  gateway-half  retransmission. 

The  argument  for  gateway-half  retransmission  is  that  the  source-to- 
destination  retransmission  may  be  quite  inefficient,  especially  if 
the  transmission  is  across  several  networks.  It  has  been  shown 
[GITMAN,  1976]  that  a hop-by-hop  acknowledgment  and  retransmission 
scheme  in  addition  to  source-to-destination  retransmission  improves 
network  efficiency,  in  particular  when  the  » mber  of  hops  from 
source-to-destination  is  large.  These  results  can  be  applied  to 
the  analogous  situation  when  considering  ETE  intranet  retransmission 
in  addition  to  the  source-to-destination  retransmission.  Moreover, 
with  gateway-half  retransmission,  the  poor  performance  of  one  network 
will  have  less  impact  on  the  performance  of  other  networks. 

C.  Checksum 

The  trade-off  in  selecting  an  appropriate  checksum  is  between 
efficiency  and  reliability:  A polynomial  checksum  provides  for  the 

protection  of  a wide  variety  of  error  conditions,  but  it  is  costly 
to  compute.  On  the  other  hand,  an  additive  type  of  checksum,  which 
may  be  easy  to  compute,  provides  a smaller  measure  of  protection. 

It  should  be  noted  that  with  present  technology,  the  hardwired 
polynomial  checksum  can  be’ produced  inexpensively.  Thus,  the  trend 
will  probably  go  toward  the  use  of  polynomial  checksums. 
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Gateway-Half  Functional  Requirements 


In  the  following,  we  list  the  basic  functions  (capabilities)  of 
gateway-halves:  [BBN,  1975],  [SUNSHINE,  1977]. 

A.  Message  Processing  Between  Two  Networks 

This  is  the  most  basic  task  a gateway-half  must  perform:  con- 

vert traffic  in  the  format  of  one  network  into  traffic  in  the  format 
of  another  network. 

B.  Inter-Gateway  Routing 


This  issue  will  be  discussed  in  Seccion  7.3.4. 

C.  Access  Control  Mechanisms 

This  function  is  desirable  to  enable  a constituent  network  to 
limit  some  classes  of  traffic.  Also,  some  users  may  impose  a con- 
straint on  passing  through  some  constituent  network.  In  general, 
the  ability  of  a network  to  monitor  the  use  of  its  resources  by  the 
internetwork  traffic  appears  to  be  desirable. 

D.  Accounting  Mechanism 


Some  form  of  accounting  on  the  internetwork  traffic  is  clearly 
necessary.  Various  types  of  accounting  are  possible,  and  the  gate- 
way-halves  seem  to  be  the  natural  place  to  do  it. 

E.  Statistics  Gathering 


The  statistics  on  the  performance  of  the  gateway-halves  and 
its  Host  networks  are  important  for  the  optimal  use  of  the  networks. 
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F . Inter n etwork  Message  Fragmentation 


The  arguments  for  or  against  message  fragmentation  at  gateway- 
halves  has  already  been  elaborated  upon  in  Sect: on  7.2.3.  At 
present,  it  seems  to  be  a desirable  option. 

G.  Congestion  Control 


This  capability  is  especially  important  because  of  the 
heterogeneous  nature  of  the  networks.  A particular  subproblem 
is  the  congestion  resulted  from  deadlocks. 

H.  Possible  Inter-Gateway  Retransmission 


The  issue  related  to  this  gateway-half  capability  has  been 
discussed  in  Section  7.2.3. 
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7 . 3 INTERNETWORK  TOPOLOGICAL  DESIGN  PROBLEM  FORMULATION  AND 

INTERNET  ROUTI NG  AND  DESIGN  ALTERNATIVES 

In  this  section  we  first  present  a formulation  of  the  general 
internetwork  topological  design  problem.  We  then  examine  some 
possible  alternatives  in  internetwork  routing  and  topological  design 
Based  on  a selection  of  the  alternatives,  we  formulate  an  internet 
routing  model  and  the  corresponding  reduced  topological  design  prob- 
lem in  the  next  two  sections. 

7.3.1  Internetwork  Topological  Design  Problem  Formulation 

Based  on  discussions  in  Section  7.2,  a formulation  for  the 
general  internetwork  topological  design  problem  can  be  given  as 
follows : 

Given : 

• Local  networks  that  participate  in 

intercommunication,  together  with 
their  Hosts  and  switches. 

• (Internetwork  and  intranetwork)  traffic 

requirements  between  the  Hosts. 

• Facility  costs/capabilities. 


Optimize : 


Total  communications  cost  D: 

D = gateway-half  cost  + intergateway 
link  cost  + local  net  link  cost  + 
local  net  switch  costs. 
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Over  the  Variables: 

• Number  and  location  of  the  gateway-halves. 

• Topology  and  link  capacities  of  the  local 
nets. 

• Gateway-half  virtual  net  topology  and  link 
capacities. 


Such  That: 


• Traffic  requirements  are  accommodated  under 
suitable  routing  strategies. 

• Average  end-to-end  delay  constraints  (may  be 
network-pair  dependent)  are  met. 

• Other  appropriate  constraints  (e.g.,  2- 
connectivity ) are  met. 

The  topological  design  problem  as  stated  above  is  very  diffi- 
cult to  solve.  For  example,  an  internet  requirement  may  be  routed 
over  several  different  netv;orks,  each  with  its  own  routing  and  flow 
control  strategy.  Moreover,  the  traffic  requirement  characteristics 
and  end-to-end  delay  constraints  may  be  network-pair  dependent  (i.e., 
dependent  upon  the  originating  and  destination  nets)  and  hence  the 
routing  problem  is  one  of  multiple  flow  classes  routing  over  multiple 
non-homogeneous  networks.  Consequently,  in  this  study  we  focus  on  a 
simplified  design  problem.  In  the  remainder  of  this  section  we 
consider  the  alternatives  in  internet  routing  and  topological  design. 
Based  on  these  considerations,  we  formulate  a simplified  internet 
routing  problem  and  the  corresponding  topological  design  problem  in 
the  next  two  sections. 
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7.3.2 


internet  Routing  and  Design  ^ternatives 


in  this  subsection  we  examine  some  of  the  important  alterna- 
tives in  internet  routing  and  topological  design:  route  selection 

alternatives,  route  restriction  alternatives,  routing  prion  Y 
alternatives,  and  local  net  design  alternatives.  These  alternatives 
form  the  basis  of  the  simplified  internet  routing  model  and 
basis  of  our  experimental  studies. 

7. 3. 2.1  Route  Selection  Alternatives^ 

This  refers  to  the  alternatives  in  selecting  a route  from  the 
source  Host  to  the  destination  Host.  A very  detailed  discussion 

_ • rcriMciiTNF  19771  The  issues  involved  are. 
can  be  found  m [SUNSHINE,  19/ /J 

1.  Should  a fixed  or  an  adaptive  routing  policy 
be  used? 

2.  Should  the  Hosts  participate  in  route  selection? 

3.  Should  a single  level  or  a hierarchical  routing 
policy  be  used? 

in  general,  the  adaptive  routing  policies  are  more  robust  and 
efficient  than  the  fixed  routing  policies.  This  is  especia  y 
true  in  the  multi-network  environment,  because  some  networ 
.ore  prone  to  congestion  and  failure  than  the  others.  However 
with  an  adaptive  policy,  individual  node's  malfunction  can  e^e"p 
into  a global  catastrophy;  consequently,  proper  safeguard  must 

implemented  at  the  gateway-halves  (or  nodes). 

Next  consider  the  issues  of  Host  routing.  At  present, 
routing  is  not  used.  However,  with  Host  routing  each  Host  can 
be  connected  to  several  different  switches,  which  is  desi 
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from  a reliability  point  of  view.  From  the  standpoint  of  inti  rnet- 
working,  Host  routing  is  also  desirable  because  the  gateway-halves 
behave  as  Hosts.  For  simplicity,  Host  routing  is  not  considered  in 
this  study. 

Next  consider  the  issue  hierarchical  levels  in  internet  routing 
With  a single  level  routing,  more  efficient  routes  can  be  selected. 
However,  this  approach  has  two  severe  drawbacks.  First,  the  routing 
table  is  necessarily  very  large,  including  all  the  nodes  in  all  nets 
Second,  the  local  net  routing  becomes  transparent  to  nodes  in  other 
networks,  which  violates  the  local  net  independence  principle.  With 
a hierarchical  routing  policy,  in  which  the  gateway-halves  (or  the 
switches  associated  with  the  gateway-halves)  act  as  local  "central 
offices,"  the  above  complications  can  be  avoided,  at  the  expense  of 
some  inefficiency.  A message  is  first  routed  from  the  source  Host 
to  a gateway-half  in  the  same  network.  It  is  then  routed  in  the 
gateway-half  virtual  network  from  to  a gateway-half  G 2 in  the 
destination  network.  (Notice  that  each  step  in  the  gateway-half  vir 
tual  net  routing  represents  either  routing  along  some  intergateway 
link,  or  pseudo-link,  i.e.,  routing  in  a local  net  with  the  gateway- 
halves  as  source  and  destination  "Hosts".)  Finally,  the  message  is 
routed  in  the  destination  network  from  G ^ to  the  destination  Host. 
For  simplicity,  we  assume  that  an  internet  message  from  (or  to)  a 
Host  is  always  routed  through  its  nearest  gateway-half  except  when 
this  gateway-half  fails.  Consequently,  each  network  is  partitioned 
by  the  resident  gateway-halves  into  disjoint  regions. 

7 . 3 . 2 . 2 Routing  Restriction  Alternatives 

There  are  various  possible  network  related  restrictions  on  the 
permissible  routes  for  requirements  of  a particular  type.  The  fol- 
lowing are  some  of  the  possibilities: 
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A . Local  Requirement  Routing  Rostrl ction 

Whether  a requirement  between  Hosts  of  the  same  network  can 
only  be  routed  in  the  same  local  net,  or  can  be  routed  through  the 
GH  virtual  net  and  other  local  nets? 

B.  Internet  Requirement  Routing  Restriction 


Whether  the  internet  requirements  between  the  gateway-halves 
(which  are  induced  from  the  Host-to-Ilost  internet  requirements)  can 
only  be  routed  on  the  intergateway  links,  or  can  be  routed  through 
local  nets? 

C . Network  Dependent  Routing  Restriction 

This  type  of  restriction  may  take  the  following  general  form: 
For  three  networks  N^,  N2  and  , the  internet  requirements  between 
Hosts  in  N^  and  N2  cannot  (or  alternatively,  must)  be  routed  through 


In  the  present  study,  only  type  A and  type  B restrictions  are 
considered.  Specifically,  we  assume  that  the  local  net  require- 
ments can  only  be  routed  in  the  same  local  net;  whereas  the  internet 
requirements  can  be  routed  over  any  nets. 

The  alternative  of  allowing  local  requirements  to  route  over 
any  net  is  interesting  in  that  constraints  of  net  boundaries  do  not 
have  to  be  taken  into  account  for  routing.  The  merits  of  such  rout- 
ing is  apparent  when  failures  of  links  and  nodes  occur  and  when  peak 
loads  in  the  different  networks  do  not  occur  simultaneously.  This 
routing  option  is  not  considered  in  this  study.  Type  C restrictions 
can  be  used  to  model  many  realistic  constraints,  however,  they  are 
difficult  to  apply  in  network  design. 
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7 . 3 . 2 . 3 Routing  Priority  Alternatives 

This  issue  refers  to  the  different  priorities  one  can  assign  to 
the  different  types  of  requirements.  The  following  are  some  of  the 
possible  options: 


1.  Local  requirements  in  net  A (A  can  be  any 
net)  have  (non-preemptive)  priority  over 
any  other  requirements  using  net  A resources. 


2.  Internet  requirements  have  priority  over 
local  requirements. 

t. 

3.  No  priority  is  assigned. . 

i 

5 

7. 3. 2. 4 Local  Net  Design  Alternatives 

In  principle,  when  both  the  local  requirements  and  the  internee 
requirements  are  using  the  resources  of  a network,  the  local  net  topo- 
logy and  link  capacities  should  be  re-optimized  to  correspond  to  the 
new  traffic  pattern.  However,  this  may  not  be  allowed  because  of 
network  ownership,  organizational,  or  security  reasons.  The  follow- 
ing are  the  possible  alternatives: 


1.  Both  the  local  net  topologies  and  the  link 
capacities  are  fixed. 

2.  The  local  net  topologies  are  fixed,  but  the 
link  capacities  can  be  modified. 

3.  The  local  net  topologies  are  allowed  to  be 
modified. 
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7.4  A SIMPLIFIED  INTERNET  ROUTING  MODEL 


In  one  context,  the  internet  routing  problem  is  that  of  rout- 
ing the  local  requirements  in  their  respective  local  nets,  and 
routing  the  internet  requirements  over  all  the  local  nets  and  inter- 
gateway links,  subject  to  the  following  delay  constraints: 

1.  Average  end-to-end  delay  constraints  for  the 
intranetwork  requirements  (can  be  different 
for  different  nets); 

2.  Average  end-to-end  delay  constraints  for  the 
internetwork  requirements  between  a pair  of 
nets  (can  be  different  for  different  pairs 
of  nets) . 

Similar  to  the  single  network  - single  packet  flow  class  case, 
one  can  develop  simple  approximate  expressions  for  the  average  end- 
to-end  delays  in  the  multi-network  context.  For  local  nets  A,  B 
and  (local  or  intergateway)  link  Z,  let 

f D(&)  = total  flow  (bits/sec)  on  link  Z due  to  re- 

A , rs 

quirements  between  Hosts  in  network  A and 
Hosts  in  network  B, 

Ta  bU)  = average  packet  delay  (sec/packet)  of  fA  fiU) 
on  link  Z, 

and 

yA  o = total  requirements  (bits/sec)  between 

A / D 

Hosts  in  network  A and  Hosts  in  net 
work  B. 
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Then  the  average  end-to-end  delay  for  YA#B  is  9iven  bY' 


lA,  B 


Y 


A,  B 


z 

all  links 


£a,b<*>  ta,b(*> 


(1) 


(£)  present, 


The  derivation  is  similar  to  [KLEINROCK,  1972] . 

On  each  link  l,  there  may  be  many  flow  groups  fAB 
and  the  average  delay  experienced  on  link  l by  each  flow  group  i- 
dependent  upon  the  interaction  of  all  the  flow  groups  on  l.  More- 
over, each  local  net  may  have  distinct  routing  and  flow  control  stra- 
tegies. Consequently,  performance  (i.e.,  end-to-end  delay)  evalua- 
tion is  a very  difficult  problem  in  this  multi-flow,  mult  -network 
context.  Equation  (1),  which  is  simple  in  appearance,  in  practice 
is  difficult  to  evaluate. 

To  alleviate  the  difficulty,  we  formulate  a simplified  internet 
routing  model  with  the  following  assumptions: 


Each  Host  is  connected  to  a unique  switch. 
This  assumption  simplifies  the  routing  prob- 
lem to  that  of  routing  between  the  switches. 


Local  requirements  are  routed  only  within 
the  same  local  net. 

The  gateway-halves  are  responsible  for  inter- 
net routing.  A gateway-half  virtual  net  rout- 
ing table  is  stored  at  each  gateway-half.  The 
routing  table  is  maintained  and  updated  in  the 
same  manner  as  the  single  net  routing  table. 

This  approach  allows  better  internet  management, 
local  net  independence  and  growth  flexibility. 

The  acknowledgment  scheme  in  the  gateway-half 
virtual  network  is  hop-by-hop.  (Thus,  for  two 
gateway-halves  in  the  same  local  net,  the  inter- 
net acknowledgment  on  the  "pseudo-link"  appears 
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as  end-to-end  acknowledgment  in  the  local  net.) 

5.  The  gateway-halves  in  the  same  local  net  parti- 
tion the  network  into  disjoint  regions.  Each 
gateway-half  serves  as  a local  internet  routing 
center  for  the  switches  in  its  region.  However, 
when  the  regional  gateway-half  is  inoperative 
the  internet  traffic  is  routed  through  a neigh- 
boring gateway-half. 

6.  For  simplicity,  each  gateway-half  resides  in  a 
switch.  Thus,  conceptually  we  can  identify  a 
gateway-half  with  the  switch  it  resides  in. 

Under  these  assumptions,  the  routing  of  an  internet  message  is 
carried  out  in  three  steps: 

1.  Local  net  routing  from  the  source  switch  (or 
Host)  to  the  source  regional  gateway-half. 

2.  Gateway-half  virtual  net  routing  from  the 
source  regional  gateway-half  to  the  destina- 
tion regional  gateway-half. 

3.  Local  net  routing  from  the  destination  re- 
gional gateway-half  to  the  destination  switch 
(or  Host) . 

Based  on  the  above  routing  model,  the  original  local  and  inter 
net  traffic  requirements  between  the  Hosts  (in  the  same  or  in  dif- 
ferent nets)  can  be  re-grouped  into  the  following  three  classes: 
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1.  Local  requirements  between  switches  in  the 
same  local  not  corresponding  to  the  orig- 
inal local  requirements; 

2.  Local  requirements  between  switches  and 
their  regional  gateway-halves  induced  by 
the  internet  requirements; 

3.  Internet  requirements  between  the  gateway- 
halves  in  different  nets  induced  by  the 
internet  requirements. 

We  call  the  above  three  types  of  requirements:  local  switch  to 

switch,  local  switch  to  gateway-half,  and  internet  gateway-half  co 
gateway-half  requirements. 

Corresponding  to  the  routing  model  and  the  resulting  traffic 
requirement  partition  introduced  in  Section  7.4.1,  we  can  consider 
three  types  of  delay  constraints: 

1.  Local  net  average  end-to-end  delay  constraint 
for  the  local  net  switch-switch  requirements 
(can  be  different  for  different  nets) ; 

2.  Average  switch  to  gateway-half  delay  constraint 
for  the  local  net  switch-gateway  half  require- 
ments (can  be  different  for  different  nets) ; 

3.  Average  end-to-end  delay  constraint  for  the 
internet  gateway-half  to  gateway-half  require- 
ments in  the  gateway-half  virtual  net. 
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By  setting  the  second  type  (switch  to  gateway-half)  constraints 
appropriately,  we  can  model  the  more  general  average  end-to-end  delay 
constraints  between  different  network  pairs.  (Admittedly  this  model- 
ing is  imperfect.)  Notice  that  the  first  two  types  of  constraints 
(and  corresponding  flows)  are  local,  and  consequently  the  routing 
and  delay  evaluation  can  be  done  for  each  local  net  separately. 

The  only  interaction  is  between  the  internet  gateway-half  to  gateway- 
half  flow  and  the  local  flows  in  each  local  net.  The  complexity 
of  the  problem  is  thus  significantly  reduced. 
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7.5  A SIMPLIFIED  INTERNETWORK  TOPOLOGICAL  DESIGN  PROBLEM  AND 
SOLUTION  APPROACH 


Based  on  the  internet  routing  model  developed  in  Section  7.4, 
we  formulate  a simplified  internet  topological  design  problem.  A 
heuristic  solution  approach  based  on  extensions  of  existing  topo- 
logical design  techniques  is  also  proposed.  Finally,  a heuristic 
procedure  for  selecting  the  gateway-halves  is  presented. 

7.5.1  Formulation  of  the  Internetwor k Topolog ical  Design 
Problem 


Based  on  the  discussions  in  the  last  section,  we  modify  the 
internetwork  topological  design  problem  given  in  Section  7.3.1  to 
be  as  follows: 

GIVEN: 

• Local  networks  that  participate  ir  inter- 

net communication,  together  with  their 
switches. 

• (internetwork  and  intranetwork)  traffic 

requirements  between  the  switches. 

• Facility  costs/capabilities. 

OPTIMIZE: 

Total  communications  cost  D: 


D = gateway-half  cost  + intergateway  link  cost  + 
local  net  link  cost  + local  net  switch  cost. 
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OVER  THE  VARIABLES: 

• Number  and  location  of  the  gateway-halves. 

• Topology  and  link  capacities  of  the  local 
nets.  (This  may  not  be  allowed  as  a vari- 
able. ) 

• Gateway -half  virtual  net  topology  and  link 
capacities. 


SUCH  THAT: 


• Traffic  requirements  are  accommodated  (under 
suitable  local  net  and  gateway-half  virtual 
net  routing  strategies) . 

• Average  end-to-end  delay  constraints  (local 
switch  to  switch,  local  switch  to  gateway- 
half,  internet  gateway-half  to  gateway-half) 
are  met. 

• Other  appropriate  constraints  (e.g.,  2-connec- 
tivity) are  met. 

7.5.2  A Solution  Approach  for  the  Internet  Topological  Design 

Problem 


Topological  design  is  in  general  a very  difficult  problem,  even 
in  the  case  of  a single  network.  Consequently,  an  effective  exact 
solution  procedure  for  the  internetwork  topological  design  problem 
formulated  in  Section  7.5.1,  appears  unlikely.  However,  based  on  ex- 
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tensions  of  existing  design  techniques  for  a single  network,  good 
heuristic  solution  approaches  are  possible.  One  approach  is  as 
follows : 


Step  1.  Find  a set  of  "good"  gateway-half  locations 
(based  on  internet  cost  tradeoff  considera- 
tions) . 

Step  2.  For  each  network,  based  on  the  gateway- 

halves  chosen,  partition  the  switches  into 
disjoint  sets. 


Step  3.  Determine  a "good"  internetwork  topology 
(i.e.,  local  net  topologies  and  inter- 
gateway connections) . 

Step  4.  For  each  local  net,  determine  the  link 
capacities  and  the  routes  for  the  local 
(type  1(a)  and  type  1(b))  requirements  so 
that  the  respective  delay  constraints  are 
satisfied.  Iso,  determine  the  average 
delay  and  average  incremental  delay  be- 
tween each  pair  of  gateway-halves  in  the 
same  net  (which  corresponds  to  the  average 
delay  and  average  incremental  delay  on  the 
gateway-half  pseudo-links ) . 


Step  5.  Determine  the  intergateway  link  capacities 
so  that  the  delay  constraints  for  the  type 
3 (internet  gateway-half  to  gateway-half) 
requirements  are  satisfied  in  the  gateway- 
half  virtual  net. 
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Step  6.  Based  on  the  internet  flow  assigned  to  the 

pseudo-links  (which  corresponds  to  additional 
requirements  between  the  gateway-halves  in 
the  same  local  net) , route  the  internet  flow 
onto  the  local  links.  Re-evaluate  the  delay 
for  the  local  requirements. 

Step  7.  If  the  stopping  criteria  (e.g.,  number  of 
iterations  or  network  cost  differences  be- 
tween iterations)  are  satisfied,  then  stop; 
otherwise,  go  to  Step  3. 

Several  remarks  are  in  order.  First,  the  above  approach  allows 
each  local  net  to  have  a different  routing  and  network  design  strat- 
egy. Second,  if  the  local  net  topologies  (and  capacity)  are  not 
allowed  to  change,  then  in  Step  3,  only  the  intergateway  links  are 
selected.  Third,  in  the  above  approach,  the  local  net  routing  is 
done  before  the  internet  routing.  This  is  based  on  the  assumption 
that  due  to  the  local  net  independence  principle,  it  is  more  important 
to  accommodate  the  local  requirements  in  the  respective  local  nets, 
especially  if  the  local  net  topologies  are  not  allowed  to  change. 

(The  internet  requirements  can  always  be  satisfied  by  adding  more 
gateway-halves  and  more  links  between  the  gateway-halves).  Finally, 
note  that  it  is  possible  for  the  local  end-to-end  delay  to  exceed 
the  constraint  after  Step  6.  In  all  of  our  design  experiments,  only 
a small  amount  of  internet  gateway-half  to  gateway-half  requirement 
is  routed  on  the  local  links,  hence  this  problem  does  not  occur. 
Moreover,  the  routing  of  intergateway  requirements  in  local  nets  can 
be  constrained  so  that  the  amount  of  internet  flow  placed  on  the 
local  links  will  never  cause  the  local  net  end-to-end  delay  constraints 
to  be  violated. 
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Step  1 of  the  solution  approach,  the  selection  of  the  gateway- 
half  locations,  will  be  discussed  in  Section  7.5.3.  In  order  to 
avoid  excessive  running  time,  we  assume  that  the  gateway-halves  are 
co-located  with  the  switches  (i.e.,  the  switch  locations  form  the 
set  of  gateway-half  location  candidates).  Steps  3 through  6 of  the 
internet  topological  design  procedure  can  be  carried  out  by  applying 
single  network  routing  and  topological  design  techniques  (e.g., 
[GERLA,  1977],  [BOORSTYN,  1977],  [CANTOR,  1974],  [MARUYAMA,  1976a], 
[MARUYAMA,  1976b])  to  the  local  nets  and  the  gateway-half  virtual 
nets.  A description  of  some  of  the  network  design  strategies  is 
given  in  Chapter  3 of  this  semiannual  report. 

7.5.3  Gateway-Half  Selection  Frocedure 

In  this  subsection  we  consider  the  problem  of  selecting  the 
gateway-half  locations  for  internetworking.  We  present  an  effective 
solution  procedure,  based  on  extensions  of  solution  techniques  for 
the  backbone  switch  location  problem  [NAC,  1976a]  and  the  satellite 
switch  location  problem  [NAC,  1976b] . 

7. 5. 3.1  Procedure  Development 

For  any  switches  a,  b in  the  same  local  net,  let 

CL(a,b)  = average  cost  of  transmitting  a unit  traffic 
requirement  from  a to  b through  the  local 
net. 

Also,  for  gateway-halves  G^,  G let 

CI(G^,G2)  = average  cost  of  transmitting  a unit  traffic 
requirement  from  G^  to  G2  through  the  gate- 
way-half virtual  net. 
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Let  Q be  a gateway-half  candidate  in  local  net  . We  examine 
the  cost  tradeoffs  involved  in  using  Q as  a gateway-half.  (By  as- 
sumption, Q is  co-located  with  one  of  the  switches  in  N^.)  In  the 
equations  that  follow,  G denotes  a gateway-half.  Let  a be  a switch 
in  N,  and  G its  associated  gateway-half.  Let  N0  be  any  other  local 
net,  b a switch  in  N2,  and  G^  the  gateway-half  associated  with  b. 

The  cost  of  transmitting  a unit  requirement  from  a to  b is  (see 
Figure  7)  ; 

C (a , b)  = CL(a,Ga)  + CI(Ga,Gb)  + CL(Gb,b).  (2) 

If  Q is  used  as  a gateway-half,  and  a is  assigned  to  Q,  then  the 
cost  for  transmission  becomes; 

C (a , b)  = CL (a , Q)  + CI(Q,Gb)  + CL(Gb,b)  (3) 

The  cost  difference  of  assigning  a to  Q,  relative  to  the  (a,b)  pair, 
is  thn.q; 

S (Q, a , b)  = [CL(a,Ga)  - CL(a,Q)J  + [CI(Ga,Gb)  - CI(Q,Gb)]. 

(4) 

For  a gateway-half  G not  in  N^,  let 

t(a,G)  = traffic  requirements  between  switch  a 
and  switches  associated  with  G. 

t(a)  = total  internet  traffic  requirement  of 

switch  a. 

Then  by  Equation  (4),  the  saving  of  assigning  a to  Q is; 


7.38 


NETWORK  ANALYSIS  CORPORATION 


G GATEWAY -HALF 

Q GATEWAY-HALF  CANDIDATE 
a SWITCH 
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S(Q,a) 


E t (a,G)  { [CL(a,G  )-CL(a,Q) j + [CI(G  ,G)  -CI(Q,G)]} 
GiNx 


= t (a)  [CL (a , G ) 

a 


CL (a,Q) ] + E t (a,G) [Cl (G  ,G)  -CI(Q,G)]. 
G*N1 


(5) 


For  ease  of  computation,  we  approximate  t(a,G)'s  by; 


t (a,G) 


t (a) 


t(G) 
tl  (N1)  ' 


(6) 


where; 


t(G)  = total  internet  traffic  for  switches 

associated  with  gateway-half  G, 


tl  (N^)  = total  internet  traffic  to  and  from 

all  nets  other  than  . 


Substituting 

S(Q,a) 


Equation  (6)  into  Equation  (5) , we  obtain; 


t (a) 
tl  (N1) 

- E 

G^N1 


{tl (N1)x[CL(a,Ga)-CL(a,Q) ] 
t(G)  x [Cl (G.G)-CI (Q,G) ] } 

a 


(7) 


The  total  saving  of  using  Q as  a gateway-half  is  thus; 

S (Q)  = £{s(Q-a>  |aeN1,S(Q,a)>0}  -CGfJ(Q),  (8) 

where 

C^U(Q)  = gateway-half  cost  at  Q. 

on 
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With  the  savings  for  each  gateway-half  candidate  determined 
by  Equation  (8),  the  gateway-halves  can  be  selected  using  an  ADD- 
type  iterative  procedure  (i.e.,  the  gateway-half  candidate  with  the 
largest  positive  saving  is  selected  as  the  next  gateway-half  [NAC, 
1976a] . The  only  remaining  task  is  to  estimate  the  unit  trans- 
mission cost  for  the  local  nets  and  the  internet  (i.e.,  the  CL(a,b)'s 
and  the  Cl (G^ ,G2 ) ' s) . 

7. 5.3. 2 Unit  Transmission  Cost  Estimate 


As  shown  in  Section  7.  5. 3.1,  the  optimal  gateway-half  location 
strategy  is  the  result  of  the  tradeoffs  between  the  local  net  access 
cost  and  the  internet  cost.  In  principle,  the  most  accurate  way  to 
select  the  optimal  set  of  gateway-halves  is  to  evaluate  the  total 
system  cost,  by  procedures  such  as  the  one  outlined  in  Steps  2-6  of 
Section  7.5.2,  for  each  candidate  set  of  qateway-halves . This 
process,  however,  is  too  time  consuming  to  be  of  practical  value. 

The  alternative  is  to  have  a simple,  yet  effective  way  to  estimate 
local  net  and  gateway-half  virtual  net  transmission  costs  associated 
with  a set  of  gateway-halves.  This  is  the  approach  taken  in  the 
backbone  transmission  cost  evaluation  for  selecting  the  backbone 
switches  or  the  satellite  switches  [NAC,  1976a] , [NAC,  1976b] . 

As  shown  in  [NAC,  1976a],  for  a given  set  of  backbone  switches, 
the  transmission  cost  for  a unit  of  requirement  between  two  switches 
a and  b can  be  effectively  estimated  by: 


U(a,b)  = CT  x Pxd(a,b)  x + CH  x x 


= ( C H — ~)  x Pxd(a,b)  x — , 

ID  p 


(9) 


where 

CT  = average  trunk  line  cost/unit  bandwidth/mile/month. 

C..  = average  trunk  line  termination  cost  (both  ends) /month. 

n 
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D = average  link  mileage. 

P = non-direct  routing  factor  (ratio  of  average  shortest 

path  length  to  direct  distance) . 

d(a,b)  = direct  distance  between  a and  b. 

b = average  protocol  overhead. 

p = average  link  utilization. 


Notice  that  on  the  right-hand-side  of  Equation  (9),  the  first 

term,  CTxPxd(a,b)  x gives  an  estimate  on  the  line  mileage 

cost  per  unit  requirement  between  a and  b;  the  second  term, 

CTT  x ■■  x gives  an  estimate  on  the  line  termination  cost 

HD  p 

per  unit  requirement  between  a and  b.  The  coefficients  C^,  and 
are  determined  by  the  facility  cost/capacity  options;  b is  deter- 
mined from  the  traffic  characteristics  and  network  technology;  D,  P, 

4-  >•  m i nvnnr  *i  mnnf  n 1 1 *r  T?vnor  1 mPnt'  n 1 rDCII  1 Q 1 

UHU  p U J.  lUXliV-U  ^ ~ ^ . — - — **-  ~ ~ *-  — — — — 

that  for  a given  data  base,  ana  a reasonable  set  of  switch  locations, 
the  values  for  P and  p are  fairly  stable. 

One  can  similarly  estimate  the  unit  transmission  cost  in  the 
gateway-half  virtual  net.  One  thus  obtains; 


(1)  For  each  local  net  N , and  switches  a,  b in  N, 

Ctt  (N)  1+b  (N ) 

ChN(a,b)  = (Ct(N)  + -15W>  x P(N)  X d(a'b)  x (10) 

where  the  functional  representation  CT(N),  C^CN),  etc.,  indicate 
that  these  parameters  are  functions  of  the  local  net  N; 


(2)  For  the  gateway-half  virtual  net,  and  gateway-halves 
Gl'  G2' 
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CH(V) 


l+b(V) 


CI(GrG2)  = (CT(V)  + “fvp)  x P(V)  x d(GrG2)  x UD 

where  V is  used  to  represent  the  gateway-half  virtual  net. 

7 . 5 . 3 . 3 Extension  to  Modularized  Gateway-halves 

Notice  that  in  the  basic  procedure  described  in  Section  7.5. 3.1, 
one  gateway-half  is  selected  at  a time.  However,  one  can  also 
regard  this  procedure  as  selecting  one  gateway-half  module  at  a 
time.  Thus,  during  the  ADD  selection  process,  once  a switch  is 
assigned  to  a selected  gateway-half  module,  it  is  deleted  from 
consideration,  but  the  location  corresponding  to  the  selected 
gateway-half  module  can  still  be  used  as  a candidate,  and  compete 
for  other  switches.  Moreover,  the  module  cost  for  a location  can  be 
made  to  be  a function  of  the  number  of  modules  already  selected  at 
that  location.  (The  interpretation  of  CQH  in  Equation  (8)  is  now 
the  gateway-half  module  cost.)  In  this  way,  the  cost/capacity  model 
can  be  made  more  realistic. 
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7 . 6 EXPERIMENTAL  RESULTS 

The  procedures  presented  in  Section  7.5  have  been  implemented  on 
a PDP-10.  Numerous  experiments  were  performed  to  study  cost/perfor- 
mance tradeoffs  in  interconnection  of  packet  switched  networks.  The 
following  two  data  bases  were  used: 

1.  A model  data  base  with  4 networks,  each 
having  8 switches. 

2.  ARPANET  and  a simplified  version  of 
AUTODIN  II. 

In  this  section  we  first  briefly  describe  the  internetwork  de- 
sign programs  developed.  We  then  examine  the  experimental  results 
on  the  4-net  sample  data  base,  ai  1 on  the  ARPANET-AUTODIN  II  data 
base. 

7.6.1  Program  Development 

A version  of  the  internetwork  routing  and  topological  design 
procedure  described  in  Section  7.5.2  has  been  implemented  on  a PDP-10, 
using  the  Rational  Fortran  language.  This  program  is  based  on  the 
existing  single  network,  single  packet  flow  class  network  design 
program  systems  at  NAC.  Program  modifications  were  required  because 
of  the  following  reasons: 

1.  Instead  of  only  one  class  of  flow,  two 

classes  of  flows  are  present  at  all  times. 

In  the  routing  of  the  local  requirements 
for  a local  net  N,  the  inernetwork  require- 
ments are  also  present  as  "fixed"  or  "passive" 
flows  (i.e.  the  internet  flow  may  be  present 
in  a local  net  link,  but  is  not  modifiable 
during  the  local  net  design  phase) , while 
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the  local  requirements  are  the  "active" 
f]ows  (i.e.,  the  routing  optimization  is 
done  on  the  local  flow  patterns) . In  the 
routing  of  the  internet  requirements  be- 
tween the  gateway-halves  in  the  gateway-half 
virtual  net,  the  flow  corresponding  to  the 
internet  requirements  are  the  "active"  flows, 
while  the  flows  corresponding  to  the  local 
requirements  (for  all  the  local  nets)  are 
the  "fixed"  or  "passive"  flows. 

2.  Program  I/O  capabilities  had  to  be  extended 
to  accommodate  several  network  files  (cne 
for  each  local  net,  plus  one  for  the  gateway-half 
virtual  net).  Depending  upon  the  user's  command, 
one  of  these  network  files  will  be  brought  in 
for  processing. 

Similar  to  the  single  network  design  system,  this  internetwork 
design  program  is  interactive:  The  routing  optimization  is  done 

automatically,  but  the  topology  selection  is  done  through  the  user- 
program  interaction. 

A version  of  the  internetwork  gateway-half  selection  procedure 
has  also  been  implemented  on  a PDP-10  using  the  Rational  Fortran 
language.  Upon  entering  the  data  base  of  nets  and  their  associated 
switches,  and  the  appropriate  cost  and  utilization  parameters,  the 
program  will  select  the  gateway-half  locations  (at  least  one  in  each 
local  net)  , and  assign  each  switch  to  its  nearest  gateway-half  in 
the  same  local  net.  The  resulting  data  base  of  gateway-halves  and 
switches  can  then  be  used  by  the  internetwork  design  program  to  obtain 
the  actual  line  layout,  capacity  assignment  and  flow  assignment. 
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An  auxiliary  program  was  also  developed  to  set  up  the  appropriate 
traffic  matrix  for  the  internetwork  design  program.  The  inputs  to 
this  program  are; 

1.  Original  (internetwork  and  intranetwork) 
traffic  requirements  between  the  switches, 

2.  Location  file  of  the  switches  and  the  gateway- 
halves,  together  with  the  switch  to  gateway-half 
association.  The  latter  is  obtained  from  the 
gateway-half  selection  program. 

The  outputs  of  this  program  are: 


1.  For  each  iocax  net,  the  local  traffic  require- 
ments between  the  switches.  This  is  the  sum  of 
the  local  switch-switch  requirements  and  the  local 
switch  co  gateway-half  requirements.  (Notice 
that  we  assume  the  gateway-halves  are  co-located 
with  the  switches. ) 

2.  For  the  gateway-half  virtual  net,  the  internet 
requirements  between  the  gateway-halves. 

7.6.2  Interconnection  of  a Model  4-Network  Data  Base 


7. 6. 2.1  Assumptions  on  the  Model  Data  Base 


To  study  the  general  behavior  of  network  interconnection,  we 
constructed  a model  data  base  with  the  following  characteristics: 
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Number  of  networks 

rr 

4 

Number  of  switches 

= 

32 

(8  in  each  net) 

Total  data  tr if  • requirements 

= 

5 

Mbps 

Average  data  pac  t text  portion 
size  (same  for.  all  nets) 

— 

740 

bits 

Local  net  packet  header  size 
(same  for  all  nets) 

= 

110 

bits 

Internet  packet  header  size 

= 

100 

bits 

RFNM  (Request  for  Next  Message) 
packet  size  (same  for  all  nets) 

= 

150 

bits 

Number  ol:  RFNM  packet/data' 
packet  (same  for  all  nets) 

= 

1 

Local  link  routing  overhead 

= 

7 

% 

(same  for  all  nets) 

Intergateway  link  routing  overhead  - 7% 


Notice  that  for  simplicity,  we  assume  that  all  the  local  nets  have 
the  same  packet  header  size,  RFNM  packet  size,  routing  overhead,  etc. 
The  packet  header  size  and  the  RFNM  size  are  believed  to  be  represen- 
tative of  the  current  ARPANET  technology.  The  networks,  the  inter- 
connection of  which  are  studied  in  this  section,  are  of  equal  size 
in  terms  of  number  of  nodes  and  throughput;  they  differ  in  topology 
and  node  locations.  This  example  was  purposely  constructed  for  study- 
ing properties  and  tradeoffs  in  interconnecting  "similar"  networks, 
in  contrast  to  the  case  of  interconnecting  ARPANET  and  AUTODIN  II. 

The  geographical  distribution  of  the  networks  is  as  shown  in 
Figure  8.  The  4 networks  approximately  span  the  continental  United 
States.  Each  network  covers  a distinct  area,  with  some  slight  over- 
lapping. 

From  the  information  given  above,  we  can  evaluate  the  average 
packet  length  and  the  average  protocol  overhead  for  the  local  pack- 
ets and  the  internet  packets  in  either  the  local  nets  or  internet 
(i.e. , gateway-half  virtual  net). 
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FIGURE  g:  INDEPENDENT  LOCAL  NET  TOPOLOGIES  MODEL  4-NET 

DATA  BASE:  THE  LOCAL  NETS  SHOWN  ARE  OPTIMALLY,  DESIGNED 
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In  deriving  the  average  packet  length  for  the  experiments  we  con- 
sidered only  information  packets  and  RFNM's.  The  average  protocol 
information  was  derived  using  packet  header  and  text  length  and 
RFNM's;  other  protocol  packets  were  not  taken  into  account.  [NAC, 
1975]  . 

Local  Packets  in  Local  Nets 


The  local  packets  refer  to  the  data  packets  for  the  original 
local  switch-switch  requirements.  For  this  traffic  we  obtained: 


[average  protocol). 
\ overhead  y 

/ average  packet  \ 
l length  J 


35% 

500  bits 


Local  Packets  in  Intergateway  Links 


By  Assumption  2 in  Section  4.1,  this  option  is  not  allowed. 
Internet  Packets  in  Local  Nets 


The  internet  packets  refer  to  the  data  packets  for  the  original 
internet  switch-switch  requirements.  In  this  case,  for  each  data 
packet,  both  local  and  internet  headers  are  required.  This  results: 

/ average  protocol^ _ 
l overhead 

average  packet 
length 

Internet  Packet  in  Intergateway  Link 


^ = 550  bits 


In  this  case,  the  only  packet  header  is  the  internet  packet 
header.  The  average  protocol  overhead  and  packet  length  are: 
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average  protocol 
overhead 

average  packet 
length 

We  make  the  following  assumptions  on  the  traffic  requirement 
distribution : 

1.  The  total  internetwork  and  intranetwork  traffic 
level  requirement  is  constant,  at  5 Mbps.  The 
internet  communication  only  impacts  upon  the 
available  communicating  Hosts,  and  upon  the  con- 
tributions from  the  available  communicating  Hosts 
(i.e.,  more  local  traffic  implies  more  contribu- 
tions from  the  Hosts  in  the  same  local  net) . 

2.  The  local  traffic  for  a local  net  is  distri- 
buted uniformly  among  the  local  switch  pairs 
(i.e.  , pair  of  switches  a and  b such  that  both 
a and  b are  in  the  same  local  net) . Each  local 
net  has  equal  amount  of  local  traffic  require- 
ment. 

3.  The  internet  traffic  is  distributed  uniformly 
among  all  the  internet  switch  pairs  (i.e., 
pair  of  switches  a and  b such  that  a,  b arc 
in  different  local  nets) . 

Table  1 lists  the  costs  and  capabilities  of  the  line  and 
hardware  options  used  in  the  internet  and  local  net  designs.  Notice 
that  modular  architecture  is  assumed  for  the  gateway-halves,  i.e., 
more  than  one  gateway-half  module  can  be  used  at  the  same  location. 
The  trunk  line  costs/capabilities  are  based  on  the  AT&T  Telpak 
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FACILITY  T"PE  CAPACITY  COST 

Trunk  Line 


Mileage  cost 

Fixed  cost  (both  ends) 

Data  rate  50  Kbps 

Switch 

Unit  cost 

Software  development 
cost 

Number  of  high  speed 
I/O  lines 

Number  of  low  speed 
I/O  lines 

Total  throughput 
(Host+terminal+S/F) 


25 

100 

1750  Kbps 


$5/mile/month 
$8 50/month 


$10 . 6K/month 

$60K/month  (for 
all  switches) 


Gateway-Half  Module 


Unit  cost 


$2K/month 


Software  development 
cost 

Number  of  high  speed 

I/O  lines  7 

Total  throughput  200  Kbps 


$15K/month  (for 
all  gateway-halves) 


TABLE  1:  FACILITY  COSTS  AND  CAPABILTIES ; FOR  THE  4-NET  DATA  BASE 
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offerings.  The  switch  costs/capabili teis  are  based  on  the  estimated 
BBN  Pluribus  IMP's.  The  gateway-half  module  costs/capabilities  are 
based  on  the  ARPANET  IMP- like  minicomputers. 


7 . 6 . 2 . 2 Design  Results  on  the  Model  Data  Ba S£ 


We  first  consider  the  situation  when  there  is  no  internet 


tl  di  11U  dilU  Utj -lvj  1 1 eaCfi  local 


ll’r  r'irl  ^ ^ 


1 r»  r>  1 f rpff  i p 


requirement.  In  this  case,  each  local  net  has  1.25  Mbps  traffic 
requirements.  We  design  each  net  optimally  (based  on  200  msec, 
average  end-to-end  delay  constraint) . The  resulting  network  con- 
figurations are  shown  in  Figure  8.  The  total  system  cost  is  approxi- 
mately $841K/mo.  (see  Table  2) . Usually,  local  networks  will  be 
operational  and  designed  without  consideration  of  internetworking. 

Next,  we  consider  the  case  when  there  is  1 Mbps  of  internet 
traffic  requirements  distributed  uniformly  among  all  the  pairs  of 
switches  (a,b) , where  a,  b are  in  different  local  nets.  (Each  local 
net  thus  has  1 Mbps  of  local  requirements  between  switches  in  the 
same  local  net) . The  gateway-half  selection  procedure  was  applied 
to  determine  the  optimum  number  and  location  of  gateway-halves  under 
different  cost  parameters.  From  among  the  solutions  we  selected 
the  following  two  sets  for  tradeoff  studies:  one  set  has  4 gateway- 

halves,  with  one  gateway-half  in  each  local  net;  the  other  set  has  8 
gateway-halves,  with  2 gateway-halves  in  each  local  net.  The  internet 
and  local  net  topologies  are  then  designed  for  each  set  of  gateway  - 
halves  based  on  the  p Bdure  presented  in  Section  7.4.2.  The 
results  are  shown  in  Figures  9 and  10.  The  cost  for  the  4-gateway- 
half  system  is  approximately  $1039K/mo.,  while  the  cost  for  the  8- 
gateway-half  system  is  approximately  $1083K/mo. (see  Table  2) . 

Interestingly,  we  found  during  the  design  experiments  that  if 
the  local  nets  are  not  allowed  to  be  modified,  then  the  internet 
requirements  cannot  sometimes  be  accommodated  in  the  local  nets. 
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FIGURE  10:  INTERNET  TOPOLOGY  WHEN  EACH  NET  HAS  TWO  GATEWAY-HALVES 

MODEL  4 -NET  DATA  BASE;  1-Mbps  INTERNET  REQUIREMENTS 
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This  leads  to  the  following  observation. 

Observation  1.  For  each  local  net,  if  less  than  half 
of  th  e swit  ches  are  _g a tew  a y-h  al  ve_s , 
then  the  local  net  link  capac ities 
must  be  increased  in  order  to  accom- 
modate the  local  net:  traffic  result- 
ing from  the  type  2 (i.e.,  switch  to 
gateway-half)  traffic  requirements. 

This  observation  can  be  explained  by  the  following  simple  cal- 
culation. There  is  1 Mbps  of  internet  requirement,  and  4 nets. 
Consequently,  each  local  net  has  (lx2)/4=.5  Mbps  internet  require- 
ments. Under  the  assumption  of  uniform  traffic  distribution,  this 
implies  that  each  switch  has  62.5  Kbps  internet  requirement.  Sup- 
pose gateway-halves  are  installed  at  less  than  half  of  the  switch 
locations,  i.e.,  there  are  at  most  3 gateway-halves  in  each  net, 
then  the  internet  requirements  for  the  remaining  switches  in  the 
same  net  generate  at  least  ( 8-3 ) x62 . 5=312 . 5 Kbps  of  local  net  require- 
ment. The  total  local  net  requirement,  excluding  the  overhead,  is 
thus  at  least  1312.5  Kbps  (1  Mbps  of  original  local  requirement  plus 
the  312.5  Kbps  internet  requirement).  The  local  net  link  capacities 
were  originally  sized  only  to  handle  1250  Kbps  of  local  net  require- 
ment. hence  more  capacity  is  needed. 

Observation  1 implies  that  if  the  local  nets  are  not  allowed 
to  be  modified,  then  a large  number  of  gateway-halves  are  needed, 
which  may  not  be  desirable. 

We  have  attempted  to  modify  the  local  net  topologies  to  improve 
overall  internet  cost.  However,  very  little  cost  improvement  is 
achieved.  In  fact,  we  were  only  able  to  improve  network  1 cost  by 
about  5%,  and  unable  to  improve  other  nets  at  all.  This  leads  to 
the  following  observation. 
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Observation  2.  Modifications  on  the  local  net  link 
capacities  are  sufficient  to  produce 
a good  global  design.  Hence,  topo- 
logical modifications  on  the  local 
nets  are  not  necessary. 

This  observation  can  be  partially  explained  as  follows: 

(a)  In  the  design,  example  considered,  for  each 
local  net,  the  local  switch-switch  traffic 
dominate  over  the  local  switch-gateway-half 
traffic.  Consequently,  the  change  in  the 
pattern  of  traffic  requirement  distribution 
is  not  drastic,  which  may  imply  that  the 
original  topology  is  still  adequate  for  the 
perturbed  traffic  distribution  pattern. 

(b)  The  traffic  requirements  are  distributed  and 
the  routing  algorithm  is  adaptive;  which  im- 
plies a certain  degree  of  insensitivity  of 
the  network  topology  (and  many  other  proper- 
ties) to  perturbations  in  the  traffic  require- 
ment pattern. 

Next,  we  increase  the  internet  requirements  to  2 Mbps,  and 
reduce  the  intranet  requirements  for  each  local  net  to  750  Kbps. 
Internet  design  procedures  were  then  applied  to  the  4-gateway- 
halves  set  and  the  8-gateway-halves  set.  The  cost  results  are 
entered  in  Table  2.  The  cost  for  the  4-gateway-halves  system  is 
approximately  $1187K/mo. , while  the  cost  for  the  8-gateway-halves 
system  is  approximately  $1283K/mo.  Again,  Observations  1 and  2 
hold. 
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Comparing  all  the  design  results  listed  in  Table  2,  we  make 
the  following  observations: 

Observation  3.  With  the  total  requirement  level  fixed, 
the  higher  the  proportion  of  internet 
requirement,  the  higher  the  total  system 
cost. 

Observation  3 is  somewhat  intuitive,  because  the  internet  require- 
ments usually  use  more  total  jystem  resources  than  the  local  require 
ments.  If  the  local  nets  are  not  allowed  to  change,  then  Observa- 
tion 3 is  trivially  true.  What  we  have  demonstrat  1 is  that  even 
when  the  local  nets  are  allowed  to  be  re-optimized,  the  . .ternet 
requirements  are  still  more  costly  than  the  local  requirements. 

Observation  4.  For  a fixed  level  of  internet  and 
intranet  requirements,  more 
gateway-halves  leads  to  a higher 
total  system  cost. 

Examining  the  component  costs,  one  finds  that  with  more 
gateway-halves : 

(a)  The  gateway-half  link  cost  increases 
significantly, 

(b)  The  link  cost  for  each  local  net  re- 
duces slightly, 

(c)  The  gateway-half  module  cost  increases 
slightly. 

Consequently,  the  increase  in  total  system  cost  when  more 
gateway-halves  are  used  is  due  mainly  to  the  increase  in  the 
gateway-half  link  cost. 
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Observation  4 can  be  explained  by  the  following  facts  on  the 
model  data  base: 

(a)  The  local  nets  are  more  or  less  regionalized. 

Thus,  the  access  cost  from  the  switches  to  the 
associated  gateway-half  is  usually  small  in 
comparison  to  the  gateway-half  to  gateway-hall 
communication  cost. 

(b)  The  internet  traffic  requirement  is  distributed 
uniformly  among  all  the  switch  pairs.  Con- 
sequently, for  any  selection  of  the  gateway-half 
set,  there  is  usually  comparable  direct  traffic 
requirements  between  any  pair  of  gateway-halves. 

Thus,  more  gateway-halves  used  implies  a mu~h 
more  costly  gateway-half  virtual  network. 

Before  changing  the  subject,  we  make  two  final  remarks  on 
Observation  4: 

(a)  Notice  that  the  gateway-half  module  cost 
constitutes  only  a very  small  part  of  the 
total  system  cost.  Consequently,  changes 
in  the  relative  cost  between  communication 
channels  and  the  gateway-half  modules  have 
only  minor  impact  on  Observation  4. 

(b)  In  the  model  data  base  very  powerful  switches 
(Pluribus-IMP  type)  are  used  and  the  unit 
switch  cost  is  fixed.  Consequently,  the 
switch  cost  does  not  enter  into  the  trade- 
offs on  the  different  number  of  gateway- 
halves.  If  the  switches  are  modularized, 

and  each  module  is  much  less  powerful,  then 
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the  switch  cost  will  impact  upon  the  trade- 
offs. In  general,  fewer  gateway-halves 
implies  more  switch  to  gateway-half  traffic 
in  the  local  net,  and  hence  higher  switch 
cost.  However,  the  general  trend  appears 
to  be  that  the  hardware  costs  reduce  at  a 
faster  ra;  e u.han  the  communicat  1 o~.  costs. 

Consequently,  the  communication  _ost  would 
dominate  over-  the  hardware  cost. 

Finally,  to  find  the  cost  differences  between  a local  nets- 
internet  system  and  a fully  integrated  system,  we  design  a fully 
integrated  packet-switched  network  on  the  32  switches,  based  on 
the  4 Mbps  of  "local  requirements"  and  the  1 Mbps  of  "internet 
requirements",  and  the  200  msc  •.  average  end-to-end  delay  constraint. 
The  resulting  topology  is  shown  in  Figure  11.  Assuming  that  no 
internet  protocol  overhead  is  needed,  the  total  cost  for  the  in- 
tegrated r /stem  is  $943.5K/mo.  Comparing  with  the  8-gateway-halves 
system,  le  cost  difference  amounts  to  only  15%  of  the  fully  inte- 
grated system  cost.  However,  the  8 gateway-halves  system  allows 
each  local  net  to  maintain  its  independence.  This  leads  to  the 
following  observation. 

Observation  5.  The  cost  for  an  interconnected  system 
is  of  the  same  order  as  an  integrated 
system  on  the  same  set  of  switches. 

We  thus  conclude  that  network  interconnection  is  an  effective 
strategy  for  merging  s ral  divergent  packet-switched  networks. 
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7.6.3  interconnection  of  ARPANET-AUTODIN  II  Backbone  Network 


7 . 6 . 3 . 1 Assumptions  on  the  Data  Base 

ARPANET 


For  this  study,  we  use  the  ARPANET  statistics  described  in 
[NAC,  1975].  The  network  configuration  is  as  shown  in  Figure  12. 
This  configuration  includes  all  the  nodes  installed  or  planned  for 
immediate  installation  as  of  May,  1975.  The  traffic  requirement 
distribution  is  based  on  information  collected  at  BBN  and  UCLA  be- 
tween May  1974  and  February  1975.  Due  to  its  slow  growth,  we  feel 
that  the  conclusions  drawn  from  the  1975  ARPANET  should  be  equally 
applicable  to  the  current  ARPANET. 

The  ARPANET  characteristics  a : summarized  as  follows: 


Number  of  switches 

= 

54 

Total  data  traffic  requirement 
(peak  hour) 

~ 

57. 

0 Kb] 

Average  data  packet  text  portion 
length 

= 

600 

bits 

Maximum  data  packet  text 
portion  length 

=s 

1000 

bits 

Packet  header  size 

= 

150 

bits 

RFNM  packet  size 

= 

150 

bits 

Average  number  of  RFNM 
packets/data  packet 

= 

1 

Routing  overhead 

= 

7% 

AUTODIN  II 

For  the  AUTODIN  II,  we  used  a simplified  model  of  an  8-node 
backbone  packet-s.  * ched  network.  The  network  configuration  is 
as  shown  in  Figure  13.  Summary  of  AUTODIN  II  characteristics 
are  as  follows: 
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FIGURE  13:  AUTODIN  II  BACKBONE  NETWORK  TOPOLOGY 
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Number  of  switches 

Total  data  traffic  require- 
ment (peak  hour) 

Average  data  pacKet  text 
portion  length 

Maximum  data  packet  text 
portion  length 

Packet  header  size 

RFNM  packet  size 

Average  number  of  RFNM 
packets/data  packet 

Routing  overhead 
Gateway-Half  Virtual  JNejt 

For  the  int  rgateway  links, 
istics : 


= 8 

= 1445  Kbps 

= 1500  bits 

= 2000  bits 
= 150  bits 

= 150  bits 

= 1 
= 7% 

we  assume  the  following  character- 


Average  data  packet  text 
portion  length 

Maximum  data  packet  text 
portion  length 

Packet  header  size 

RFNM  packet  size 

Average  number  of  RFNM 
packts/data  packet 

Routing  overhead 


= 600  bits 

= 1000  bits 
= 150  bits 

= 150  bits 

= 1 
= 7% 


Notice  that  in  the  gateway-half  virtual  net,  we  assume  the 
f low  control  is  done  on  a link-by-link  basis.  Thus,  RFNM  packets 
and  acknowledgments  are  required  between  each  two  adjacent 
gateway-halves. 

The  ARPANET- AUTODIN  II  data  base  is  significantly  different 
from  the  model  4-net  data  base  considered  in  Section  7.6.2.  Follow- 
ing are  the  main  differences: 
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(a)  ARPANET  and  AUTODIN  II  each  roughly  span  the 
entire  continental  United  States.  For  the  4-net 
model,  each  net  covers  a distinct  area  of  the 
continental  United  States. 

(b)  ARPANET  has  a large  number  of  switches,  but  very 
low  level  of  traffic.  AUTODIN  II  has  very  few 
backbone  switches,  but  the  traffic  level  is  high. 

Hence,  the  interconnection  of  these  two  very  dis- 
similar networks  poses  an  interesting  problem. 

For  the  4-net  model,  each  net  was  comparable  in 
size  and  traffic  magnitude. 

(c)  The  switch  distribution  for  both  ARPANET  and 
AUTODIN  II  are  concentrated  on  the  East  coast  and 
the  West  coast.  The  switch  distribution  for  thc- 
4-net  model  is  roughly  uniform. 

(d)  Neither  ARPANET  nor  AUTODIN  II  were  designed 
optimally  for  their  respective  throughput  level. 
Consequently,  both  have  spare  capacities  for  addi- 
tional traffic.  For  the  model  4-net  data  base, 

each  net  was  optimally  designed  for  the  local  traffic. 

Below  we  evaluate  the  average  packet  lengths  and  the  average 
protocol  overheads  for  the  local  packets  and  the  internet  packet 
in  the  local  nets  and  the  gateway-half  virtual  net. 

Local  ARPANET  packets  in  ARPANET 


The  local  packets  refer  to  the  data  packets  for  the  original, 
local  switch-switch  requirements.  For  this  data  base  we  obtained: 
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(average  protocol], 
overhead  J 

(average  packet  \ 
length  J 


50% 


450  bits 


Local  AUTODIN  II  packets  in  AUTODIN  II 


The  local  packets  refer  to  the  data  packets  for  the  original 
local  switch-switch  requirements: 


(average  protocol^ 
overhead  y 

(average  packet  j 
length  J 


20% 

900 


Internet  Packets  in  ARPANET  and  AUTODIN  II 


The  internet  packets  refer  to  the  data  packets  for  the  original 
requirements  between  switches  in  ARPANET  and  AUTODIN  II.  In  this 
case,  both  local  and  internet  headers  are  required: 


(average  protocol 
overhead  J 

^average  packet  \ = 
length  ) 


75% 

525  bits 


Internet  Packets  in  the  Intergateway  Links 


In  this  case,  the  only  packet  header  is  the  internet  packet 
header : 


average  protocol 
overhead 
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(average  packet 
length 

We  make  the  following  assumptions  on  the  traffic  distribution: 

1.  The  intranet  requirements  are  fixed,  i.e.,  the 

total  local  ARPANET  throughput  is  set  at  57  Kbps, 
while  the  total  AUTODIN  II  throughput  is  set  at 
1445  Kbps.  This  is  done  because  the  ARPANET  through- 
put level  is  so  low,  it  is  not  meaningful  to  dis- 
tribute the  total  ARPANET  throughput  among  local 
and  internet  requirements.  Moreover,  to  reflect 
the  realistic  environment,  it  is  probably  not  mean- 
ingful to  scale  up  the  ARPANET  traffic  level. 

2.  The  internet  traffic  requirement  between  ARPANET 
and  AUTODIN  II  is  distributed  uniformly  among  all 
the  switch  pairs  (i.e.,  pair  of  switches  a and  b, 
with  a in  ARPANET  and  b in  AUTODIN  II) . 

The  facility  costs/capabilities  are  the  same  as  that  for  the 
model  4-net  design  (see  Table  1),  except  that  for  ARPANET,  regular 
IMP'S  are  used,  with  a cost  of  $2K/mo. , and  total  throughput  of  250 
Kbps.  The  Pluribus  IMP'S  arc  used  as  the  AUTODIN  II  switches. 

7.6.4.  Design  Results  on  ARPANET-AUTODIN  II  Data  Base 

Two  basic  sets  of  experiments  were  conducted  on  the  intercon- 
nection of  ARPANET  and  AUTODIN  II.  In  the  first  set,  the  topologies 
and  link  capacities  of  both  the  ARPANET  and  AUTODIN  II  are  assumed 
fixed.  For  each  selected  set  of  gateway-halves,  w^  find  the  maxi- 
mum amount  of  internet  traffic  requirement  that  can  be  channeled 
through  the  gateway-halves  before  either  ARPANET  or  AUTODIN  II  be- 
come congested.  In  the  second  set,  we  examine  the  cost  differences 
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between  desings  allowing  local  net  modifications  and  designs  not 
allowing  local  net  modification. 

j Maximum  Inter n et  Thro ughput  vs.  Number  of  Gateway - 1 1 alves  Used 

I 

1 

Suppose  that  in  internet  communication,  the  topologies  and 
link  capacities  of  neither  ARPANET  nor  AUTODIN  II  are  allowed  to 
be  modified.  This  is  very  likely  to  be  the  situation  for  most 
internet  communication.  It  is  then  of  interest  to  find  out  the 
maximum  amount  of  internet  requirement  that  can  bo  accommodated 
for  a given  set  of  gateway-halves.  We  have  performed  numerous 
experiments  on  different  sets  of  gateway-halves  in  ARPANET  and 
AUTODIN  II,  and  the  results  are  summarized  in  Figure  14.  For  con- 
venience, we  used  the  total  number  of  gateway-halves  in  both  nets 
as  the  parameter.  It  was  found  that  for  all  of  the  designs 
considered,  ARPANET  capacity  is  always  the  limiting  factor  in 
internet  throughput.  In  Figure  15,  we  plot  the  maximum  internet 
throughput  versus  the  number  of  gateway-halves  used  in  ARPANET.  It 
can  be  seen  from  both  Figures  14  and  15  that  the  internet  throughput 
level  is  slightly  concave  with  respect  to  the  number  of  gateway- 
halves  used.  Moreover,  with  just  a few  gateway-halves,  an  internet 
throughput  level  of  100-300  Kbps  can  be  easily  achieved. 

Total  Syste m Cost  Comparison  Between  Allowing  Local  Net  Modification 
and  Not  Allowing  Local  Net  Modification 

Suppose  both  ARPANET  and  AUTODIN  II  are  allowed  to  be  modi- 
field.  Then  intuitively,  for  the  same  internet  throughput  level, 
the  total  system  cost  should  be  lower  than  when  the  local  nets  are 
not  allowed  to  be  modified.  In  Figure  16  we  plot  the  total  system 
costs  for  both  cases  with  the  internet  throughput  level  as  the  para- 
meter. The  cost  versus  throughput  curves  were  also  shown  for  dif- 
ferent unit  component  costs.  Examining  Figure  16,  one  can  observe 
that; 
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INTERNET  THROUGHPUT 
(Kbps) 


NUMBER  OF 
GATEWAY-HALVES 
IN  ARPANET 


FIGURE  15:  INTERNET  THROUGHPUT  VS.  NUMBER  OF  GATEWAY-HALVES  USED  IN  ARPANET; 

ARPANET-AUTODIN  I.I  DATA  BASE;  FIXED  LOCAL  NETS 
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For  the  same  unit  communications  cost,  (i.e., 
trunk  line  mileage  cost) , the  difference  in 
total  system  cost  between  designs  with  local 
nets  fixed  and  designs  with  local  nets  change- 
able is  nearly  constant  regardless  of  the  total 
internet  throughput  level. 


The  difference  in  total  system  cost  between 
the  two  types  of  designs  is  quite  sensitive  to 
changes  in  unit  communications  cost,  and  is 
less  sensitive  to  changes  in  unit  gateway-half 
module  cost. 


Suppose  the  unit  gateway-half  module  cost  is 
expensive  (for  example,  $ 1 OK/mo/module , ) as  is 
one  of  the  cases  considered  in  Figure  16. 

Then  the  cost  differences  between  designs  with 
local  nets  fixed  and  designs  with  local  nets 
changeable  increases  as  the  internet  throughput 
level  increases.  This  can  be  explained  as  follows 
If  the  local  nets  are  fixed,  then  as  the  internet 
throughput  level  increases,  more  gateway-halves 
are  needed,  as  i . shown  in  Figures  14  and  15. 

On  the  other  hand,  if  the  local  nets  are  modi- 
fiable, then  the  same  internet  throughput  level 
can  be  achieved  with  fewer  gateway-halves  by 
doing  some  local  net  modifications.  Consequently, 
with  expensive  gateway-half  modules,  the  cost 
differences  between  the  two  types  of  designs  in- 
creases with  the  internet  throughput  level. 
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Since  one  can  predict  with  reasonable  confidence  that  both  the 
unit  communications  cost  and  the  unit  hardware  cost  will  continue 
to  decline , the  above  observations  indicate  that  the  cost  penalty 
paid  by  requiring  local  nets  fixed  in  internetworking  will  ecome 
! less  and  less  significant.  At  the  base  unit  cost  used  in  his  study  - 

$ 5/mile/month  for  the  50  Kbps  trunk  lines  and  $2K/month  for  a 
gateway-half  module  with  200  Kbps  throughput  (which  are  estimates  of 
the  current  market  prices)  - the  extra  cost  paid  by  the  fixed  local 
net  designs  amounts  to  less  than  8%  of  the  total  system  cost. 
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7.7  CONCLUSIONS 


In  this  chapter  the  topological  design  problem  of  interconnect- 
ing pac’et  switched  networks  was  studied. 

The  gateway-half  model  was  adopted  as  the  medium  of  internet 
connection.  Using  the  gateway-halves  as  the  higher  level  nodes, 
a hierarchical  internet  routing  model  ras  developed.  A reduced  in- 
ternet topological  design  problem  was  en  formulated.  A solution 
approach,  using  extensions  of  existing  uechniques , . was  also  presented. 
The  solution  procedure  was  implemented  in  a PDP-10,  and  was  applied 
to  study  the  interconnection  problem  in  a multi-network  model,  and 
the  interconnection  of  ARPANET  and  AUTODIN  II. 

The  following  observations  emerge  from  the  studies  of  interconnect- 
ing 4 equal  size  networks: 

1.  The  higher  the  proportion  of  internet 
requirement,  the  higher  the  total  sys- 
tem cost. 

2.  If  neither  the  local  net  topology  nor 
link  capacities  are  allowed  to  change, 
then  gateway-halves  are  needed  at  at 
least,  half  of  the  switch  locations. 

3.  If  only  a moderate  number  of  gateway- 
halves  are  used,  then  good  internet  de- 
s±gn  can  be  obtained  by  allowing  modi- 
fications on  the  local  link  capacities. 

4.  The  cost  for  an  interconnected  packet- 
switched  system  is  of  the  same  order  as 
the  cost  for  an  integrated  packet-switched 
system,  on  the  same  set  of  switches. 
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On  the  ARPANET- AUTODIN  II  i terconncction  study,  the  assumption 
was  that  the  local  traffic  requirements  are  fixed.  Studies  were  then 
carried  out  to  determine  the  maximum  internet  throughput  level  for 
various  number  of  gateway-halves.  Parametric  studies  on  the  cost  com- 
parison between  designs  with  local  nets  fixed  and  designs  with  local 
nets  changeable  were  also  carried  out.  It  was  found  that  at  the  base 
unit  cost  ($5/mile/mo.  for  50Kbps  trunk  lines  and  $2K/mo.  for  the 
gateway-half  modules) , the  cost  differences  amounts  to  only  5-10%  of 
the  total  system  cost.  It  was  also  found  out  that  the  cost  differences 
are  more  sensitive  to  changes  in  the  unit  communications  cost,  and  less* 
sensitive  to  changes  in  the  unit  gateway-half  module  cost. 

Internet  topological  design  is  a new  area  of  research  (a;*  ’ r. 
the  entire  area  of  network  interconnection) . Hence,  there  are 
many  issues  open  for  further  investigation.  We  indicate  a few  of 
the  relevant  issues  here: 

1.  General  hierarchical  routing  strategies:  The 

internet  model  developed  in  this  paper  assumes 
a very  special  hierarchical  structure:  Each 

Host  is  associated  with  a unique  switch,  and 
each  switch  is  associated  with  a unique  gate- 
way-half in  the  same  network.  In  a more  gen- 
eral context,  each  Host  (switch,  respectively) 
can  associate  with  several  switches  (gateway- 
halves,  respectively)  in  the  same  net.  This 
allows  more  flexible  and  reliable  host-switch 
connection  and  internet  connection,  and  better 
flow  and  congestion  control.  Consequently, 
hierarchical  routing  st^tegies  which  allow 
each  node  to  select  one  of  several  higher 
level  nodes  as  its  "homing  nodes"  are  desirable. 
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2.  Local  net  design  considerations  in  the  multi- 
network context:  The  local  net  design  approach 

used  in  this  study  takes  only  the  local  (switch 
to  switch,  switch  to  gateway-half)  requirements 
into  account.  Thus,  the  local  net  design  and 
gateway-half  virtual  net  design  are  essentially 
two  separate  problems.  This  approach  is  similar 
to  many  of  the  national  telephone  systems: 

Usually  a certain  number  of  cross-country  chan- 
nels between  international  gateways  are  reserved 
for  international  traffic;  these  channels  can  be 
regarded  as  capacity  taken  away  from  the  resident 
national  net.  However,  other  local  net  design 
strategies  with  better  capacity-sharing  scheme 
for  the  local-internet  traffic  may  be  more  de- 
sirable from  the  cost-effectiveness  viewpoint. 

3.  General  multi-network  interconnection  and  inte- 
gration: The  problem  addressed  in  this  paper 

concerns  only  the  interconnection  of  packet- 
switched  networks.  However,  generally,  the 
packet-switched  networks  serve  as  the  back- 
bones for  large  distributed  networks  in  a 
multi-level  architecture.  Consequently,  the 
interconnection  and/or  integration  problem 
should  be  most  appropriately  addressed  in  the 
total  system  context. 
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